1. Field of the Invention
This invention relates generally to the field of computer graphics and, more particularly, to high performance graphics systems which generate images for display on one or more display devices, including projection devices.
2. Description of the Related Art
Multiple projection devices may be used to display a single integrated image as shown in FIG. 1. Each of the projection devices may display a portion of the integrated image. For example, projection devices P1 and P2 may target a common display screen SCR, and may generate the left and right portions respectively of an integrated image. The projection devices may use any of a variety of technologies. For example, the projection devices may be based on cathode ray tubes (CRTs), liquid crystal displays (LCDs), digital micro-mirror displays, or liquid crystal on silicon (LCOS) displays, etc.
A computer system CS may be used to drive the multiple projection devices P1 and P2. The computer system CS may provide analog or digital video signals to drive the projection devices P1 and P2. The computer system CS may include a graphics system for the rendering and display of 2D graphics and/or 3D graphics. The graphics system may supply the video signals which drive the projection devices P1 and P2. In addition, the computer system CS may include a monitor device MD, a system unit SU, and input devices such as a keyboard KB and mouse MS. Monitor device MD and projection devices P1 and P2 may be referred to generically as display devices.
It is often necessary to overlap the images generated by the two projection devices P1 and P2 as shown in FIG. 2A. A front view of the two overlapped images as they might appear on the display screen SCR is shown in FIG. 2B. The left image represented by rectangle ABCD overlaps the right image represented by rectangle EFGH.
FIG. 2C illustrates a fundamental problem associated with the overlapping of two projected images. The function I1 represents the distribution of light intensity across the screen SCR due to projection device P1. Similarly, function I2 represents the distribution of light intensity across the screen SCR due to projection device P2. In the overlap region (i.e. rectangle EBCH) the intensity functions add. Thus, the intensity in the overlap region may be larger in the non-overlap regions than in the non-overlapped regions of the screen. The increased brightness of the overlap region may be annoying to a viewer of the integrated image AFGD.
The overlap-brightness problem exists when any number of projected images are overlapped. For example, FIG. 3 illustrates the overlapping of four projected images generated by four projection devices in a xe2x80x9ctwo by twoxe2x80x9d configuration. One of the projected images is represented by rectangle JLTR. The other three projected images have the same size as rectangle JLTR and are displaced to the three remaining corners M, V and Y. Exactly two of the projected images overlap in regions KLPO, NOSR, STXW and PQUT, and thus, these regions may exhibit a light intensity which is double that of the non-overlap regions JKON, LMQP, RSWV and TUYX. All four projected images overlap in the center region OPST, and thus, the center region may exhibit a light intensity which is four times larger than the intensity of the non-overlap regions.
There currently exist analog systems called xe2x80x9cvideo boxesxe2x80x9d which address in some measure the overlap-brightness problem. A video box may couple between computer system CS and projection devices P1 and P2 as shown in FIG. 1, and may reduce the intensity of the projected images in the overlap region. Thus, the total intensity in the overlap region may be more nearly equal to the intensity of the non-overlap regions after installation of a video box. Although, video boxes provide some measure of correction to the overlap-brightness problem, they may be expensive and may not be easy to configure. Furthermore, their functionality may be limited. Thus, there exists a need for a system and method which could provide an improved blending of the light intensity in the overlap regions. More generally, there exists a need for a system and method which could uniformize the intensity distribution of an integrated image generated by multiple projection devices.
In FIG. 2C, the intensity distributions I1 and I2 are illustrated as having rectangular profiles. In reality, the intensity distribution of a projected image may be highly non-uniform for a number of different reasons. First, a typical projection device PD radiates light with a non-uniform radiation intensity pattern as shown in FIG. 4A. The radiation intensity pattern may have a maximum in the on-axis direction, i.e. the direction of central projection ray CPR, and may decrease as a function of the radiation angle RA measured with respect to the central projection ray CPR.
Thus, as shown in FIG. 4B, a more realistic screen intensity distribution for a projected image may have a maximum at the point CM where the central projection ray intercepts the screen SCR, and may decrease as a function of horizontal and vertical screen displacements from the central maximum CM. Several level curves of the screen intensity distribution are superimposed on the projected image. FIG. 4C illustrates the screen intensity distribution ID along a horizontal slice HS of the screen through the central maximum CM.
If the screen material is highly diffusive, the screen intensity distribution ID may be relatively independent of the viewing angle and/or position. Diffusivity means that a generic ray R impinging on the screen is scattered with a hemispherical scattering distribution SD. However, if the screen material has high gain, i.e. is more reflective (or transmissive in the rear-projection case), the screen intensity distribution may be highly dependent on the viewer""s position with respect to the projector-screen system.
For example, FIG. 5A illustrates the screen intensity distributions IV1 and IV2 perceived by viewers V1 and V2 respectively situated on opposite sides of projection device PD in a front-projection scenario. The screen intensity distribution IV1 for the first viewer V1 may attain a maximum value at the point P1 where the source ray S1 emanating from the projection device has a reflection ray R1 which intercepts the viewer V1. Thus, the viewer V1 perceives a xe2x80x9chot spotxe2x80x9d on the screen at point P1. Similarly, the screen intensity distribution IV2 for the second viewer V2 may attain a maximum value at the point P2 where the source ray S2 emanating from the projection device has a reflection ray R2 which intercepts the viewer V2. Thus, viewer V2 perceives a hot spot at point P2 on the screen SCR.
FIG. 5B illustrates the screen intensity distributions IV1 and IV2 perceived by viewers V1 and V2 respectively situated at different positions with respect to a high gain screen in a rear-projection scenario. The screen intensity distribution IV1 for the first viewer V1 may attain a maximum value at the point Q1 where the direct ray D1 from the projection device to the viewer V1 intercepts the screen. Similarly, the screen intensity distribution IV2 for the second viewer V2 may attain a maximum value at the point Q2 where the direct ray D2 from the projection device to the viewer V2 intercepts the screen.
Multiple projector devices may be used to generate an integrated image on the projection screen SCR. Thus, an observer may observe two or more hot spots on the screen, i.e. the intensity distribution may have two or more local maxima. Thus, there exists a need for a system and method which could correct the hot spots perceived by the observer on screen SCR.
Projection devices generally attempt to project a rectangular array of physical pixels on the projection screen SCR. However, many distortion mechanisms conspire to defeat this desired end. For example, the physical pixel array may exhibit a keystone distortion KST, a pincushion distortion PNC or a barrel distortion BRL as suggested by FIG. 6A. Fundamental distortion modes may combine to form more complex distortions as suggested by combined distortion CDST.
A keystone distortion is synonymously referred to herein as a trapezoidal distortion. As used herein the term xe2x80x9ckeystone distortionxe2x80x9d is intended to include any distortion where the nominally rectangular pixel array is deformed into a trapezoid. The parallel edges of the trapezoid may be either the top and bottom edges, or, the right and left edges. The terms xe2x80x9cpincushion distortionxe2x80x9d and xe2x80x9cbarrel distortionxe2x80x9d are terms which are familiar from fields such as optics and video projection. Thus, the exact definition of these terms may depend on the context in which they are used. As used herein, the term xe2x80x9cpin-cushion distortionxe2x80x9d is intended to include any distortion where the right and left edges, and/or, the top and bottom edges, of the nominally rectangular pixel array experience an inward bending as presented on screen SCR. For example, a CRT display distortion where the right and left sides are bent inwardly while horizontal lines remain relatively straight is referred to herein as a pincushion distortion. As another example, a lens-induced distortion where all four edges are bent inwardly is also referred to herein as a pincushion distortion. As used herein, the term xe2x80x9cbarrel distortionxe2x80x9d is intended to include any distortion where the right and left edges, and/or, the top and bottom edges, of the nominally rectangular pixel array experience an outward bending. For example, a CRT distortion where the right and left sides are bent outwardly, and a lens-induced distortion where all four sides are bent outwardly, are both referred to herein as barrel distortions.
Distortions of the displayed pixel array may arise from a variety of physical mechanisms and have a large variation in magnitude. For example, in a scenario where a projection device projects onto a screen which is not centered on or perpendicular to the projection axis, the resulting geometric distortion may amount to a significant fraction of the displayed pixel array. Other distortions of the displayed pixel array may be subtle, i.e. small in magnitude. Thus, there exists a need for a graphics system and methodology which could compensate for arbitrary physical pixel distortions so that rendered objects would appear undistorted on the screen SCR.
As noted above, multiple projection devices may be used to generate an integrated image or integrated sequence of video images on screen SCR as shown in FIG. 1. In general, the physical pixel arrays generated by the projection devices are overlapped as suggested by the example of FIG. 6B. The left physical array LPA (denoted by small circles) is generated by a first projection device, and the right physical array RPA (denoted by small triangles) is generated by a second projection device. Each physical array may be distorted in an arbitrary manner. Thus, images displayed in the left and right physical arrays may appear distorted. For example, the left physical array LPA is illustrated with a barrel distortion, and the right physical array is illustrated with a keystone (i.e. trapezoidal) distortion.
Even if the distortion of each physical array is small, there may be relative distortions between the left and right physical arrays. For example, the left and fight physical arrays may not have the same vertical and/or horizontal size, or one physical array may be translated with respect to the other. As another example, the distortions of the left and right physical arrays may arise from manufacturing imperfections in the light processing chips of each projection device. Thus, images (or portions of images) appearing in the overlap region may appear blurred (i.e. fuzzy). Thus, there exists a need for a graphics system which could compensate for arbitrary physical distortions (including relative distortions) inherent in multiple projection configurations so that the integrated image generated by the multiple projection devices may appear undistorted and consistent, especially in overlap regions. In other words, there exists a need for a system and method which could correct for the independent physical distortions arising in multiple projector scenarios.
The area covered by the left physical array LPA and right physical array RPA on the screen SCR is the effective viewing area. In order to maximize the viewing area, it would be desirable to minimize the size of the overlap between the left physical array LPA and the right physical array RPA. However, if the overlap region is minimized, it becomes difficult to perform edge blending with the prior art video box technology. Thus, there may appear an uneven line of brightness along the interface of the physical arrays. The brightness is due to a higher density of physical pixels. Thus, there exists a need for a system and method which could neatly match the boundaries of physical images generated by multiple projection devices configured with minimum overlap and without generating a line of bright distortion along the interfaces.
Modem projection devices generate color images by a variety of different mechanisms which lead to different modes of chromatic distortion. As used herein the term xe2x80x9cchromatic distortionxe2x80x9d is intended to include any distortion which results in a spatial separation of displayed color components of physical pixels on a screen. The following is a discussion of several economically viable projection technologies and their chromatic distortion modes.
According to one model, a projection device may comprise three separate color beams which are projected through three respective lenses (or lensing systems) onto the screen SCR. For example, a CRT-based projector may have separate red, green and blue displays, i.e. three separate guns targeting corresponding phosphor surfaces. Each display generates a source image of the corresponding color which is projected onto the screen SCR through a corresponding lens. The projection of each source image onto screen SCR is referred to herein as a physical image. For example, the red source image from the red display is projected onto a red physical image on screen SCR. Ideally, the red, green and blue physical images on screen SCR would exactly overlay one another, generating a single rectangular array of pixel positions. Corresponding pixels in each of the three physical images would cover the same area on screen SCR. However, real projection devices depart from this ideal behavior because of distortions inherent in the displays, the lenses, the alignment of the displays with respect to the lenses, the alignment of the lenses with respect to the screen, etc. Thus, the red, green and blue physical images are independently distorted on SCR, and corresponding pixels of the three physical images fail to overlap on screen SCR.
According to a second model, a projection device may include a single embedded display (e.g. an integrated LCD display) and an optical subsystem. The embedded display generates a source image comprising an array of source pixels. The optical subsystem projects the source image onto screen SCR. Each source pixel may radiate a combination of red, blue and green light spectra. Because optical materials (such as lenses) in the optical subsystem bend light rays in a wavelength dependent fashion, a single source pixel may be fragmented into separate red, green and blue physical pixels on screen SCR. This effect is known as chromatic aberration. Chromatic aberration is responsible for a blurring of images on screen SCR.
According to a third model, a high performance projector may use dichroic optics as follows. A single source beam of white light may be separated into three subbeams. The three subbeams are independently modulated with color information resulting in a red subbeam, a green subbeam and a blue subbeam. For example, the three white subbeams may be transmitted through three corresponding LDC panels, or refracted from three corresponding digital micromirror chips. The red, green and blue subbeams may be combined using dichroic mirrors, and the combined beam may be projected through a single lens onto screen SCR. Under ideal conditions, corresponding pixels in the modulated red, green and blue subbeams would be aligned in the combined beam. However, in actual practice, one may observe independent distortions and separation of the red, green and blue pixels in the combined beam because of various manufacturing imperfections. For example, the cells in each light processing chip may be independently distorted away from a nominal rectangular array. There may be slight variations in the alignment and pathlength calibration of the light processing chips with respect to the dichroic optics, etc. The distortions and separation of the red, green and blue pixels in the combined beam may be exacerbated as the combined beam passes through the lens and is projected onto screen SCR due to chromatic aberration (i.e. wavelength dependent refraction). Thus, the red, green and blue physical images on screen SCR are independently distorted away from an ideal rectangle, and corresponding pixels in the three physical images fail to overlap. Furthermore, the relative displacements between corresponding pixels in the three physical images are likely to vary in magnitude and direction across the screen.
According to a fourth model, a projection device may employ a single light processing chip (such as a digital micromirror chip) with a color modulation device (such as a color wheel) to generate a single output beam bearing all three colors in a time multiplexed fashion. The human visual system performs the color integration. The single output beam is projected onto the screen SCR through a single lens (or lensing system). The red, green and blue pixels in the single output beam are distorted due to chromatic aberration in the lens. However, because the source light beam is not split and recombined, many of the distortion mechanisms of the third model above are avoided.
According to a fifth model, a projection device may employ two light processing chips. A source light beam may be split into two subbeams. The first subbeam may be modulated with color information for a first color using the first light processing chip. The second subbeam may be modulated with the color information for the two remaining colors using the second light processing chip and a color wheel. The two modulated subbeams may be recombined using dichroic optics. The combined beam is then projected onto the screen SCR through a single lens (or lens system). The red, green and blue pixels in the combined beam experience chromatic aberration in passage through the lens. However, prior to interaction with the lens, the distortions and separation of the red, green and blue pixels in the combined beam may have magnitudes that are less than in the third (three-way split and recombine) model and larger than in the fourth (no split) model.
Thus, there exists a need for a system and method which could compensate for arbitrary chromatic distortions (i.e. distortions which separate the color components of a composite physical image) independent of the mechanisms by which such distortions arise within a projection device.
Computer system CS may rely on a graphics system to generate output video signals for projection devices P1 and P2 (and/or monitor device MD). Early graphics systems were only responsible for receiving data produced by the host processor and displaying it on monitor device MD. In essence, they acted as simple translators or interfaces. Modern graphics systems, however, incorporate graphics processors with a great deal of processing power. They now act more like coprocessors rather than simple translators. This change is due to the recent increase in both the complexity and amount of data being sent to monitor device MD and/or the projection devices. For example, modem display devices have many more pixels, greater color depth, and are able to display more complex images with higher refresh rates than earlier models. Similarly, the images displayed are now more complex and may involve advanced techniques such as anti-aliasing and texture mapping.
As a result, without considerable processing power in the graphics system, the host processor would spend a great deal of time performing graphics calculations. This could rob the host processor of the processing power needed for performing other tasks associated with program execution, and thereby dramatically reduce the overall performance of computer system CS. With a powerful graphics system, however, when the host processor is instructed to draw a box on the screen, the host processor may send a request to the graphics system stating xe2x80x9cdraw a box at these coordinates.xe2x80x9d The graphics system may then compute the position and color of each pixel comprising the box. The host processor is thus liberated from the time consuming draw or render process.
Generally, a graphics system is a type of video adapter that contains its own processor to boost performance levels. These processors are specialized for computing graphical transformations, and thus, they tend to achieve better results than the general-purpose CPU used by computer system CS. In addition, they free up the computer""s CPU to execute other commands while the graphics system is handling graphics computations. The popularity of graphical applications, and especially multimedia applications, has made high performance graphics systems a common feature of computer systems. Most computer manufacturers now bundle a high performance graphics system with their computer systems.
Since graphics systems typically perform only a limited set of functions, they may be customized and therefore are far more efficient at graphics operations than the computer""s general-purpose CPU. While early graphics systems were limited to performing two-dimensional (2D) graphics, their functionality has increased to support three-dimensional (3D) wire-frame graphics, 3D solids, and now includes support for three-dimensional (3D) graphics with textures and special effects such as advanced shading, fogging, alpha-blending, and specular highlighting.
Prior art graphics systems use a conventional frame buffer for refreshing pixel/video data on a display device (e.g. monitor device MD, or projection device P1 or P2). The frame buffer stores rows and columns of pixels that exactly correspond to respective row and column locations on the display device. Prior art graphics system render 2D and/or 3D images or objects into the frame buffer in pixel form, and then read the pixels from the frame buffer during a screen refresh to refresh the display device. Thus, the frame buffer stores the output pixels that are provided to the display device. To reduce visual artifacts that may be created by refreshing the screen at the same time the frame buffer is being updated, frame buffers in most graphics systems are double-buffered.
To obtain more realistic images, some prior art graphics systems have gone further by generating more than one sample per pixel. As used herein, the term xe2x80x9csamplexe2x80x9d refers to information calculated at a particular point on an object or image, i.e. information such as one or more of color, depth (z), transparency, etc. For example a sample may comprise the following component values: a red value, a green value, a blue value, a z value, and an alpha value (e.g., representing the transparency of the sample). A sample may also comprise other information, e.g., a z-depth value, a blur value, an intensity value, brighter-than-bright information, and an indicator that the sample partially or completely comprises control information rather than color information (i.e., xe2x80x9csample control informationxe2x80x9d). By calculating more samples than pixels (i.e., super-sampling), a more detailed image is calculated than can be displayed on the display device. For example, a graphics system may calculate four samples for each pixel to be output to the display device. After the samples are calculated, they are then combined or filtered to form the pixels that are stored in the frame buffer and then conveyed to the display device. Using pixels formed in this manner may create a more realistic final image because overly abrupt changes in the image may be smoothed by the filtering process.
These prior art super-sampling systems typically generate a number of samples that are far greater than the number of pixel locations on the display. These prior art systems typically have rendering processors that calculate the samples and store them into a render buffer. Filtering hardware then reads the samples from the render buffer, filters the samples to create pixels, and then stores the pixels in a conventional frame buffer. The conventional frame buffer is typically double-buffered, with one side being used for refreshing the display device while the other side is updated by the filtering hardware. These systems, however, have generally suffered from limitations imposed by the conventional frame buffer, and by the added latency caused by the render buffer and filtering hardware. Therefore, an improved graphics system is desired which includes the benefits of pixel super-sampling while avoiding the drawbacks of the conventional frame buffer.
U.S. patent application Ser. No. 09/251,844 titled xe2x80x9cGraphics System with a Variable Resolution Sample Bufferxe2x80x9d discloses a computer graphics system that utilizes a super-sampled sample buffer and a sample-to-pixel calculation unit for refreshing a display device. The graphics processor generates a plurality of samples and stores them into a sample buffer. The graphics processor preferably generates and stores more than one sample for at least a subset of the pixel locations on the display device. Thus, the sample buffer is a super-sampled sample buffer which stores a number of samples that may be far greater than the number of pixel locations on the display device. The sample-to-pixel calculation unit is configured to read the samples from the super-sampled sample buffer and filter or convolve the samples into respective output pixels, where the output pixels are then provided to refresh the display device. The sample-to-pixel calculation unit selects one or more samples and filters them to generate an output pixel. The sample-to-pixel calculation unit may operate to obtain samples and generate output pixels which are provided directly to the display device with no frame buffer therebetween.
Thus, a system and method is desired for providing improved display capabilities including improved edge blending, hotspot correction, edge matching, distortion correction, and chromatic distortion compensation in a system having a super-sampled sample buffer.
A graphics system according to the present invention may comprise a graphics processor, a super-sampled sample buffer, and one or more sample-to-pixel calculation units. The graphics processor generates a plurality of samples and stores them into the sample buffer. Each sample-to-pixel calculation unit operates on sample values to generate pixel values. The graphics processor preferably generates and stores more than one sample per pixel for at least a subset of the pixel locations on a display surface. Thus, the sample buffer is a super-sampled sample buffer which stores a number of samples that, in some embodiments, may be far greater than the number of pixel locations on the display surface. In other embodiments, the total number of samples may be closer to, equal to, or even less than the total number of pixel locations on the display surface. In addition, the the samples may be more densely positioned for certain areas of the display surface and less densely positioned in other areas.
The sample-to-pixel calculation unit is configured to read the samples from the super-sampled sample buffer, and to filter the samples in order to generate respective output pixel values. The output pixel values are then provided to refresh a display device (e.g. monitor or projection device). The sample-to-pixel calculation unit selects one or more samples and filters them to generate an output pixel value. Note the number of samples selected and or filtered by the sample-to-pixel calculation unit may be greater than one (as in the preferred embodiment), equal to one, or less than one.
The sample-to-pixel calculation unit may access the samples from the sample buffer, perform a real-time filtering operation, and then provide the resulting output pixel values to the display device, preferably in real-time. The graphics system may transmit computed pixel values to a display device without storing them in an intervening frame buffer. Thus, the large latency of a frame buffer may be avoided. It is noted that some display devices may have internal frame buffers, but these are considered an integral part of the display device, not the graphics system. Thus, the sample-to-pixel calculation units may calculate each pixel for each video frame on a real time basis or on an on-the-fly basis.
In one embodiment, the graphics system may be configured to perform edge blending by flattening the distribution of light energy perceived by a viewer of two or more images generated on a display surface by two or more corresponding projection devices. The graphics system may comprise a first sample-to-pixel calculation unit and a second sample-to-pixel calculation unit. The first sample-to-pixel calculation unit may be configured to generate a first plurality of first virtual pixel positions in a two-dimensional viewport which is populated with samples, to compute first pixel values by filtering the samples at the first plurality of first virtual pixel positions, to scale a first subset of the first pixel values by first intensity scaling values, and to provide the first pixel values to a first projection device. The second sample-to-pixel calculation unit may be similarly configured to generate a second plurality of second virtual pixel positions in the two-dimensional viewport, to compute second pixel values by filtering the samples at the second plurality of second virtual pixel positions, to scale a second subset of the second pixel values by second intensity scaling values, and to provide the second pixel values to a second projection device.
The first projection device may be configured to generate a first array of first physical pixels on the display surface in response to the first pixel values. Similarly, the second projection device may be configured to generate a second array of second physical pixels on the display surface in response to the second pixel values. The first subset of the first pixel values which are scaled using the first intensity scaling values may control intensities of a first subset of the first physical pixels in a first overlap region between the first array and the second array. The second subset of the second pixel values which are scaled using the second intensity scaling values may control intensities of a second subset of the second physical pixels in the first overlap region.
The first intensity scaling values may transition from one to zero from a first side of the first overlap region to a second side of the first overlap region. Conversely, the second intensity scaling values may transition from zero to one from the first side of the first overlap region to the second side of the first overlap region. The first intensity scaling values may transition linearly (piecewise linearly, non-linearly, etc.) from one to zero from the first side of the first overlap region to the second side of the first overlap region. Conversely, the second intensity scaling values may transition linearly (piecewise linearly, non-linearly, etc.) from zero to one from the first side of the first overlap region to the second side of the first overlap region.
The first intensity scaling values and the second intensity scaling values may be assigned numeric values so that the total light intensity in the first overlap region is approximately equal to the light intensity in the non-overlap regions of the first array and the second array. The first intensity scaling values may be computed based on a first intensity scaling function, and the second intensity scaling values may be computed based on a second intensity scaling function. Preferably, the first intensity scaling function and the second intensity scaling function add up to one on the first overlap region. The intensity scaling described above in terms of two projection devices may be generalized to an arbitrary number of projection devices whose screen images overlap, perhaps with multiple overlappings.
In an alternative embodiment, the graphics system comprises a first calculation unit and a second calculation unit. The first calculation unit may be configured to generate first pixel values, and to provide the first pixel values to a first projection device. The first projection device may be configured to generate a first array of first physical pixels on a display surface. The second calculation unit may be configured to generate second pixel values, and to provide the second pixel values to a second projection device. The second projection device may be configured to generate a second array of second physical pixels on the display surface. The first physical pixel array and the second physical pixel array overlap in a first overlap region on the display surface. The first pixel values control intensities of first physical pixels. The second pixel values control intensities of the second physical pixels. The first calculation unit is configured to multiply a first subset of the first pixel values which correspond to the first overlap region by first intensity scaling values prior to providing the first pixel values to the first projection device. The second calculation unit is configured to multiply a second subset of the second pixel values which correspond to the first overlap region by second intensity scaling values prior to providing the second pixel values to the second projection device. In this alternative embodiment, the first calculation unit and second calculation unit may compute pixel values in response to 3D graphics data by any conventional means.
In a second embodiment, the graphics system may be configured to perform hotspot correction by flattening the distribution of light energy perceived by a viewer of a displayed image generated on a display surface by one or more display devices. The graphics system may comprise a first sample-to-pixel calculation unit configured to generate a first plurality of first virtual pixel positions in a two-dimensional viewport which is populated with samples, to compute first pixel values by filtering the samples at the first plurality of first virtual pixel positions, to multiply the first pixel values by first intensity scaling values to produce first scaled pixel values, and to provide the first scaled pixel values to a first projection device. The first intensity scaling values may be configured to correct for a non-uniformity (e.g. a hotspot) in the perceived distribution of light intensity by a viewer.
The first sample-to-pixel calculation unit may be configured to determine the viewer""s perceived distribution (of light intensity from the screen surface SCR) in response to received sensor data. The sensor data may be received from a position sensing device located at the viewer (e.g. on the user""s head). The viewer""s position in the projector-screen geometric system may be used to compute the viewer""s perceived distribution based on known and/or measured optical properties of the projection device and screen SCR. Furthermore, the viewer""s perceived distribution may be used to generate the first intensity scaling values. For example, the first sample-to-pixel calculation unit may be configured to normalize a minimum of the viewer""s distribution to one, to invert the normalized distribution, and to evaluate the inverted normalized distribution in order to generate the first intensity scaling values. The first sample-to-calculation unit may be configured to repeatedly receive the sensor data, and to repeatedly generate the first intensity scaling values based on the sensor data as the viewer moves with respect to the display surface.
The first intensity scaling values may be between zero and one inclusive, and may be used to attenuate a first subset of the first pixel values which map onto a first hot spot region of the display surface. A first subset of the first intensity scaling values may be stored in a memory, and a second subset of the first intensity scaling values may be interpolated based on the first subset of the first intensity scaling values stored in the memory.
The graphics system embodiment described above may be generalized to operate with any number of projection devices. For example, the graphics system may further comprise a second sample-to-pixel calculation unit configured to generate a second plurality of second virtual pixel positions in the two-dimensional viewport, to compute second pixel values by filtering the samples at the second plurality of second virtual pixel positions, to multiply the second pixel values by second intensity scaling values to produce second scaled pixel values, and to provide the second scaled pixel values to a second projection device. The first intensity scaling values and the second intensity scaling values may be configured to correct the non-uniformity in the perceived distribution of light by the viewer. The second intensity scaling values may be configured to attenuate the second pixel values in the neighborhood of a second hot spot generated by the second projection device.
In a third embodiment, the graphics system may be configured to correct arbitrary display distortions, i.e. to compensate for arbitrary distortions of physical pixels (i.e. displayed pixels) on a display surface regardless of the mechanisms by which these distortions arise. The graphics system may comprise a graphics processor, a sample buffer, and a first sample-to-pixel calculation unit. The graphics processor may be operable (a) to receive graphics data, (b) to select sample positions in a two-dimensional viewport, and (c) to compute sample values at the sample positions based on the received graphics data. The sample buffer may be coupled to the graphics processor and configured to store the sample values. The first sample-to-pixel calculation unit may be coupled to the sample buffer and operable to select a first plurality of first virtual pixel positions in the two-dimensional viewport. For each first virtual pixel position in the first plurality, the first sample-to-pixel calculation unit may be operable (d) to access a first subset of the sample values corresponding to a first neighborhood of the first virtual pixel center, (e) to filter the first subset of the sample values to generate a first pixel value, and (f) to provide the first pixel value as output to a first display device. The first sample-to-pixel calculation unit may select the first plurality of first virtual pixel positions in the two-dimensional viewport in order to correct for a physical distortion of a first set of physical pixels generated by the first display device on a display surface.
The physical distortion of the first set of physical pixels may be determined relative some nominal set of screen positions (e.g. a rectangular array, a hexagonal array, etc.). The physical distortions treated by the methodologies herein disclosed may have arbitrary magnitudes. In other words, subtle physical distortions on the order of a fraction of a pixel distance may be compensated as well as large physical distortions that cover a large number of pixel distances. While small distortions on the order of a fraction of a pixel distance may not be noticeable to the viewer in a single mono-video projection scenario, such distortions have a detrimental impact on the visual quality of overlapping video and stacked video from multiple projectors. The treatment of overlapping video and stacked video with be described below.
The first sample-to-pixel calculation unit may be configured to receive calibration signals from a user/operator, and to adapt the first plurality of first virtual pixel positions in the 2-D viewport in response to the calibration signals. The first sample-to-pixel calculation unit may be configured to adapt the first plurality of first virtual pixel positions in the 2-D viewport along a one parameter family of geometric distortions in response to the calibration signals. The first sample-to-pixel calculation unit may be configured to expansively (and/or contractively) adapt the first plurality of first virtual pixel positions.
The first sample-to-pixel calculation unit may be further configured to receive sensor data indicating the physical positions of the first set of physical pixels on the display surface, and to compute the first plurality of first virtual pixel positions in response to the sensor data.
A manufacturer of display devices (e.g. monitors or projection devices) may calibrate the first virtual pixel positions for the first display device at manufacture time. The calibrated first virtual pixel positions for the first display device may be stored on a memory medium internal or external to the first display device. The manufacturer may ship the memory medium to a user along with the first display device. The user may transfer the calibrated first virtual pixel positions from the memory medium to the graphics system. The calibrated first virtual pixel positions may allow the user to immediately (or more quickly) achieve undistorted presentation of images using the graphics system and the first display device. It is noted that the need for manufacture-time calibration depends on the magnitude of variations in display distortions from unit to unit
A trained worker at a manufacturing facility may calibrate the first virtual pixel positions by selecting one or more calibration modes (e.g. keystone mode, barrel mode, pincushion mode, etc.), and supplying plus/minus adaptation inputs for each calibration mode, until displayed objects appear undistorted. Alternatively, the manufacturer""s calibration may be automated. For example, an adaptive control system may be configured to replace the trained worker.
In addition to, or perhaps, instead of, manufacture-time calibration, the graphics system may provide a mechanism (such as those suggested above) for user calibration of the first virtual pixel positions. The need for a user calibration mechanism may depend on the magnitude of time-variation in the display distortions.
If the time variation of display distortions is sufficiently large, the graphics system may be configured for continuous (e.g. real-time) calibration. For example, the graphics system may be configured to intermittently display subliminal calibration patterns via the first display device. A video camera may monitor the presentation of the calibration patterns on the first display device (or screen SCR). The graphics system may analyze the video images, and may generate corrected first virtual pixel positions in the 2-D viewport in response to the video images. Such continuous calibration may facilitate the use of high-drift display devices which would otherwise be disturbing to viewers.
The present invention contemplates various embodiments of the graphics system where any combination of manufacture-time calibration, user calibration and/or continuous calibration may be supported.
The first sample-to-pixel calculation unit may include a first cache memory. The first cache memory may be configured to store a first number of memory bins, where the first number of memory bins depends on a maximum expected displacement of the first plurality of first virtual pixel positions from corresponding positions in a rectangular array.
The first sample-to-pixel calculation unit may select the first plurality of first virtual pixel positions with a first distortion pattern which is geometrically similar (or approximately similar) to the physical distortion of the first set of physical pixels on the display surface.
In addition, the graphics system may be configured to correct for arbitrary distortions of the physical pixels generated by multiple display devices. Thus, the graphics system may also include a second sample-to-pixel calculation unit coupled to the sample buffer and operable to select a second plurality of second virtual pixel positions in the two-dimensional viewport. For each second virtual pixel position in the second plurality, the second sample-to-pixel calculation unit may be operable (g) to access a second subset of the sample values corresponding to a second neighborhood of the second virtual pixel center, (h) to filter the second subset of the sample values to generate a second pixel value, and (i) to provide the second pixel value as output to a second display device. The second sample-to-pixel calculation unit may select the second plurality of second virtual pixel positions in the two-dimensional viewport in order to correct for a physical distortion of a second set of physical pixels generated by the second display device on the display surface. The first sample-to-pixel calculation unit and second sample-to-pixel calculation unit may be configured to receive calibration inputs from a user, and to respectively adapt the first virtual pixel positions and the second virtual pixel positions in the 2-D viewport. The first and second display devices may be projection devices.
The second sample-to-pixel calculation unit may select the second virtual pixel positions with a second distortion pattern which is geometrically similar (or approximately similar) to the physical distortion of the second set of physical pixels on the display surface.
The first sample-to-pixel calculation unit and second sample-to-pixel calculation unit may be configured to adapt the first virtual pixel positions relative to the second virtual pixel positions in the 2-D viewport in response to the calibration inputs. For example, the first sample-to-pixel calculation unit and second sample-to-pixel calculation unit may be configured to translate and/or scale (vertically and/or horizontally) the first virtual pixel positions relative to the second virtual pixel positions in the 2-D viewport in response to the calibration inputs.
A manufacturer of multi-projector (or multi-display) display systems may calibrate the virtual pixel positions for each projection device in a display system at manufacture time. The set of calibrated virtual pixel positions for each projection device in the display system may be stored in a memory medium internal to or external to the projection device. The manufacturer may ship the memory medium to a user along with the display system. The user may transfer each set of calibrated virtual pixel positions from the memory medium to the graphics system. The sets of calibrated virtual pixel positions may allow the user to immediately (or more rapidly) achieve undistorted presentation of integrated images on screen SCR using the graphics system to drive the multiple projection devices comprising the display system.
In addition to, or perhaps, instead of, manufacture-time calibration, the graphics system may provide a mechanism for user calibration of the virtual pixel positions for each projection device. The need for a user calibration mechanism may depend on the magnitude of time-variation in the physical display distortions.
If the time variation of physical display distortions is sufficiently large, the graphics system may be configured for continuous (e.g. real-time) calibration. For example, the graphics system may be configured to intermittently display subliminal calibration patterns via the multiple projection devices. A video camera may monitor the presentation of the calibration patterns generated on screen SCR by the multiple projection devices. The graphics system may analyze the video images, and may generate a corrected set of virtual pixel positions for each projection device in response to the video images.
The present invention contemplates various embodiments of the graphics system where any combination of manufacture-time calibration, user calibration and/or continuous calibration may be supported.
In a fourth embodiment, the graphics system may be configured to perform edge matching (i.e. to align the boundaries of images generated by two or more projection devices on a display surface). The graphics system may comprise a first sample-to-pixel calculation unit and a second sample-to-pixel calculation unit. The first sample-to-pixel calculation unit may be configured (i) to select a first plurality of first virtual pixel positions in a two-dimensional viewport, (ii) to generate a first stream of first pixel values by filtering samples, distributed in the two-dimensional viewport, at the first plurality of first virtual pixel positions, and (iii) to provide the first stream of first pixel values to a first projection device. The first projection device may be configured to generate a first physical pixel array on the display surface in response to receiving the first stream of first pixel values.
The second sample-to-pixel calculation unit may be configured (iv) to select a second plurality of second virtual pixel positions in the two-dimensional viewport, (v) to generate a second stream of second pixel values by filtering the samples, distributed in the two-dimensional viewport, at the second plurality of second virtual pixel positions, and (vi) to provide the second stream of second pixel values to a second projection device. The second projection device may be configured to generate a second physical pixel array on the display surface in response to receiving the second stream of second pixel values. A first physical barrier may be positioned normal to the display surface and along the interface of the first physical pixel array and the second physical pixel array. The first physical barrier may be a thin strip of light absorbing material. The first physical barrier may be in physical contact with the display surface (e.g. a projection screen).
The first sample-to-pixel calculation unit may select a subset of the first plurality of first virtual pixel positions in the two-dimensional viewport in order to correct for a first physical distortion of the first physical pixel array on the display surface. Similarly, the second sample-to-pixel calculation unit may select a subset of the second plurality of second virtual pixel positions in the two-dimensional viewport in order to correct for a second physical distortion of the second physical pixel array on the display surface. Subsets of the first and second virtual pixel positions may be selected along the edge of the first and second pixel arrays to perform edge matching.
The first projection device and the second projection device may be configured so that the overlap of the first physical pixel array and the second physical pixel array is less than a predetermined maximum value. In particular, the first projection device and the second projection device may be configured so that the first physical pixel array and the second physical pixel array overlap minimally.
The first sample-to-pixel calculation unit and second sample-to-pixel calculation unit may be configured to receive calibration signals from a user, and to adapt a subset of the first plurality of first virtual pixel positions and a subset of the second plurality of second virtual pixel positions respectively in the two-dimensional viewport in response to the calibration signals. The first sample-to-pixel calculation unit and second sample-to-pixel calculation unit may be configured to translate (vertically and/or horizontally) and/or scale (vertically and/or horizontally) the first virtual pixel positions relative to the second virtual pixel positions in the two-dimensional viewport in response to the calibration inputs.
In addition, the first sample-to-pixel calculation unit and second sample-to-pixel calculation unit may be configured to receive sensor data indicating physical positions of physical pixels comprising the first physical pixel array and the second physical pixel array on the display surface; and to compute the first virtual pixel positions and the second virtual pixel positions respectively in response to the sensor data.
The first sample-to-pixel calculation unit may select the first plurality of first virtual pixel positions with a first distortion pattern which is geometrically similar (or approximately similar) to a first physical distortion expressed by the first physical pixel array. Similarly, the second sample-to-pixel calculation unit selects the second plurality of second virtual pixel positions with a second distortion pattern which is geometrically similar (or approximately similar) to a second physical distortion expressed by the second physical pixel array.
In one embodiment, the graphics system may be configured to compensate for chromatic distortion mechanisms (such as chromatic aberration). Thus, the first sample-to-pixel calculation unit may be configured to use a separate set of virtual pixel positions for each color. The first sample-to-pixel calculation unit may select NC sets of virtual pixel positions in the two-dimensional viewport, where NC corresponds to the number of colors used by a display device. For each virtual pixel position in the Kth set, the first sample-to-pixel calculation unit is operable to access samples in the neighborhood of the virtual pixel position, to filter the neighborhood samples to generate a corresponding pixel value, and to provide the pixel value as output to the display device. The pixel values generated in response to the Kth set of virtual pixel positions control the Kth color of the NC colors. The first sample-to-pixel calculation unit may select the NC sets of virtual pixel positions in the two-dimensional viewport in order to correct for arbitrary chromatic distortions arising in the display device.
Each set of virtual pixel positions may be computed in response to sensor data indicating physical positions of the physical pixels on the display surface when excited with the corresponding color. The first sample-to-pixel calculation unit selects each set of virtual pixel positions to be geometrically similar to the measured physical pixel positions when the first display device is excited only with the corresponding color. Alternatively, each set of virtual pixel positions may be computed in response to calibration inputs.